• Numpy arrays and basic element-wise operations (addition, exponentiation, etc.)
• Broadcasting arrays to perform operations on arrays of different shapes
• Indexing and slicing arrays
• Basics of Pandas Series and DataFrames
  • Creating Series and DataFrames
  • Indexing and slicing
  • Filtering
  • Sorting
• Reading and writing data from/to files

• Reshape datasets using melt() and pivot()
• Aggregate data using groupby()
• Apply functions to groups of data with apply()
• Combine data from different sources with merge()
• Handle missing data with dropna() and fillna()
• Plot data with pandas’ built-in plotting functions

• Pandas .melt(), .pivot() and .pivot_table() can help reshape dataframes

• .melt(): make wide data long

• .pivot(): make long data wide

• .pivot_table(): same as .pivot() but can handle multiple indexes

• “Wide” data has many columns and few rows

• “Long” data has many rows and few columns

• The choice of wide vs. long format depends on the analysis you want to do, but long format is often more flexible

• The data below shows how many courses different instructors taught across different years
• If the question you want to answer is something like: “Does the number of courses taught vary depending on year?”, then the data would not be considered tidy
• Why? Because there are multiple observations of courses taught in a year per row (i.e., there is data for 2018, 2019 and 2020 in a single row)

import numpy as np
import pandas as pd

df = pd.DataFrame({"Name": ["Tom", "Mike", "Tiffany", "Varada", "Joel"],
                   "2018": [1, 3, 4, 5, 3],
                   "2019": [2, 4, 3, 2, 1],
                   "2020": [5, 2, 4, 4, 3]})
df

• Let’s make it tidy with .melt()
• .melt() takes a few arguments, most important is the id_vars which indicated which column should be the identifier

df_melt = df.melt(id_vars="Name",       # identifier
                  var_name="Year",      # new column for the years
                  value_name="Courses") # new column for the courses
df_melt

• Sometimes, you may want to make long data wide
  • Examples: in presentations (for clarity)
  • Running MANOVA or repeated measures analyses
  • Building correlation matrices
• We can do that with .pivot()
• When using .pivot() we need to specify the index to pivot on, and the columns that will be used to make the new columns of the wider dataframe

df_pivot = df_melt.pivot(index="Name",
                         columns="Year",
                         values="Courses")
display(df_pivot)

• You’ll notice that Pandas set our specified index as the index of the new dataframe
• It also preserved the label of the columns
• We can easily remove these names and reset the index to make our dataframe look like it originally did with .reset_index()

df_pivot = df_pivot.reset_index() 
df_pivot

• Hmmm, what about Year? 🤔
• When pivoting, pandas sets the columns index name to match the columns parameter in .pivot(), which was Year

df_pivot.columns.name = None
df_pivot

• Often you’ll work with multiple dataframes that you want to stick together or merge
• df.merge() and df.concat() are [all you need] to know for combining dataframes
• The Pandas documentation is very helpful for these functions, but they are pretty easy to grasp
• df.concat() is used to stick dataframes together, either by rows or columns
• df.merge() is used to combine dataframes based on a common column
  • It’s like a SQL join for those familiar with SQL (if you’re not, don’t worry we’ll learn about it soon 😊)

• You can use pd.concat() to stick dataframes together:
• Vertically: if they have the same columns, OR
• Horizontally: if they have the same rows

df1 = pd.DataFrame({'A': [1, 3, 5],
                    'B': [2, 4, 6]})
df2 = pd.DataFrame({'A': [7, 9, 11],
                    'B': [8, 10, 12]})
df1

df2

• Let’s stick these dataframes together vertically
• Just pass a list of dataframes to pd.concat()

df_concat = pd.concat([df1, df2])
df_concat

• Notice that the indexes were simply joined together?
• This may or may not be what you want
• To reset the index, you can specify the argument ignore_index=True:

df_concat = pd.concat([df1, df2], ignore_index=True)
df_concat

• You can also concatenate dataframes horizontally
• If you do this, you can specify the axis argument to be 1 (default is 0)

df_concat = pd.concat([df1, df2], axis=1)
df_concat

• You are not limited to just two dataframes, you can concatenate as many as you want

pd.concat([df1, df2, df1, df2], axis=0, ignore_index=True)

• pd.merge() gives you the ability to “join” dataframes using different rules
  • Again, just like with SQL if you’re familiar with it
• You can use df.merge() to join dataframes based on shared key columns
• Methods include:
  • inner join: only keep rows where the key exists in both dataframes
  • outer join: keep all rows from both dataframes
  • left join: keep all rows from the left dataframe
  • right join: keep all rows from the right dataframe

• See this great cheat sheet and these great animations for more insights

• There will be times when you want to apply a function that is not built-in to Pandas
• For this, we also have methods:
  • df.apply(), applies a function column-wise or row-wise across a dataframe (the function must be able to accept/return an array)
  • df.applymap(), applies a function element-wise (for functions that accept/return single values at a time)
  • series.apply()/series.map(), same as above but for Pandas series
• Let’s go back to the dataset we saw before (about cycling) and see how we can apply a numpy function to it

df = pd.read_csv("data/cycling_data.csv")
df[['Time', 'Distance']].apply(np.sin)

• Or you may want to apply your own custom function

def seconds_to_hours(x):
    return x / 3600

df[['Time']].apply(seconds_to_hours)

• However, this could be done more easily with a lambda function

df[['Time']].apply(lambda x: x / 3600)

• Often we are interested in examining specific groups in our data
• df.groupby() allows us to group our data based on a variable(s)

df = pd.read_csv('data/cycling_data.csv')
df.head()

• Let’s group this dataframe on the column Name:

dfg = df.groupby(by='Name')
dfg

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x169176780>

• What is a DataFrameGroupBy object? It contains information about the groups of the dataframe

• You can access the groups using the .groups attribute
• This will return a dictionary where the keys are the group names and the values are the indices of the rows in the original dataframe that belong to that group

dfg.groups

{'Afternoon Ride': [0, 2, 4, 6, 9, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32], 'Morning Ride': [1, 3, 5, 7, 8, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]}

• We can also access a group using the .get_group() method:

dfg.get_group('Afternoon Ride').head()

Functions

• You can pass many functions to groupby objects, such as:
  • mean()
  • sum()
  • count()
  • std()
  • min()
  • max()
  • median()
  • describe()
  • apply()
  • transform()
  • filter()

dfg['Distance'].mean()

Name
Afternoon Ride    12.462
Morning Ride      12.860
Name: Distance, dtype: float64

Aggregating multiple columns

• You can also pass many functions at the same time
• The .agg() method is used for this
• It takes a dictionary where the keys are the columns and the values are the functions to apply
• Multiple functions can be passed as a list

dfg.agg({'Distance': ['mean', 'sum', 'count'], 
         'Time': ['mean', 'sum', 'count']})

• And even apply different functions to different columns

def num_range(x):
    return x.max() - x.min()

dfg.agg({"Time": ['max', 'min', 'mean', num_range], 
         "Distance": ['sum']})

• By the way, you can use .agg() for non-grouped dataframes too
• This is pretty much what df.describe() does under the hood:

# Select only numeric values
numeric_df = df.select_dtypes(include='number')

# Apply the function
numeric_df.agg(['count', 'mean', 'median', 'std', 'min', 'max'])

• Missing data is a common problem in data analysis
• Pandas has a few methods to deal with missing data
• The most common are:
  • isnull(): returns a boolean mask where True indicates missing values
  • notnull(): opposite of isnull()
  • dropna(): removes missing values
  • fillna(): fills missing values with a specified value
  • interpolate(): fills missing values with interpolated values

• We can use df.isnull() to find missing values in a dataframe

df.isnull().head(3)

• df.info() is often more useful

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 33 entries, 0 to 32
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Date      33 non-null     object 
 1   Name      33 non-null     object 
 2   Type      33 non-null     object 
 3   Time      33 non-null     int64  
 4   Distance  31 non-null     float64
 5   Comments  33 non-null     object 
dtypes: float64(1), int64(1), object(4)
memory usage: 1.7+ KB

• The simplest way to deal with missing values is to drop them
• You can use the .dropna() method to do this

df.dropna().head()

• Or you can impute (“fill”) them using .fillna()
• This method has various options for filling
• You can use a fixed value, the mean of the column, the previous non-nan value, etc

df = pd.DataFrame([[np.nan, 2, np.nan, 0],
                   [3, 4, np.nan, 1],
                   [np.nan, np.nan, np.nan, 5],
                   [np.nan, 3, np.nan, 4]],
                  columns=list('ABCD'))
df

• Cumulative sum of the Distance column

df['Distance'].cumsum().plot.line()

• You can also plot multiple columns at once

df[['Distance', 'Time']].plot()

• There are many configuration options for these plots which build of the matplotlib library

df['Distance'].cumsum().plot.line(fontsize=14, linewidth = 2, color = 'r', ylabel="km")

• You can even use custom themes with the style argument

import matplotlib.pyplot as plt
import mplcyberpunk
plt.style.use("cyberpunk")

df['Distance'].plot.line(ylabel="km")
mplcyberpunk.add_glow_effects()